Comparative Evaluation Studies of Mobile Systems

Established approaches to design and evaluate usable systems are challenged by systems for wearable, handheld, or mobile devices. There are extensive methods and guidelines that describe how the usability of stationary computer systems should be evaluated [4,6]. This is complemented with experimental evaluations of the relative strengths and weaknesses of different techniques [1,3]. However, in relation to design and evaluation of mobile systems, such methods and guidelines are yet to be produced. Mobile systems are typically used in highly dynamic contexts and their use often involves several people distributed in the user’s physical surroundings. Therefore, field-based testing seems like an appealing or even indispensable approach for evaluating mobile systems. Yet usability testing in the field is challenging. Firstly, it can be complicated to establish realistic studies that capture the richness of the use-context. Secondly, it is far from trivial to apply established techniques such as observation and think-aloud in the field. Thirdly, field-testing complicate data collection and limits control since users are moving physically in an environment with a number of unknown variables. When usability tests are conducted in a laboratory setting, control and collection of high quality data is not a problem, but one of the drawbacks is the lack of realism. Existing approaches to laboratory-based usability testing of stationary computer systems try to solve this problem by recreating or imitating the real use context, e.g. by office furnishing [6]. However, when mobile systems are tested in a laboratory setting, activities in the user’s physical surroundings can be difficult to recreate realistically [5]. We explore laboratory and field-based approaches to usability evaluation of mobile systems through a number of comparative usability studies involving different experimental design. Two of these studies are illustrated below.